This pull request introduces significant improvements to the context management and finite state machine (FSM) infrastructure in the codebase, with a focus on making context propagation more robust for asynchronous jobs, enhancing FSM reliability and configurability, and introducing new state choices for annotation drafts. The most notable changes include the addition of a CurrentContext class for thread-local context management, improved context capture for async jobs, a more flexible state transition registration decorator, and optimizations for FSM state transitions and cache handling.

Context Management and Async Job Improvements:

• Introduced the CurrentContext class in core/current_request.py to provide a unified, thread-local context management interface, including methods to set/get user, organization, and request, as well as utilities for FSM feature toggling and context clearing. All context access is now routed through this class, replacing direct thread-local usage.
• Enhanced async job handling in core/redis.py to automatically capture and serialize relevant context (user, organization, custom job data) and inject it into job metadata, ensuring context is preserved for background tasks. [1] [2]
• Updated job execution logic to cleanly separate context management for async (Redis) and sync jobs, and improved argument handling for job submission. [1] [2] [3]

FSM Infrastructure Enhancements:

• Added a feature flag check to enable or disable FSM logic dynamically, using the new context management utilities for user/organization scoping. FSM state retrieval now respects this flag, and logging includes organization context. [1] [2] [3] [4]
• Improved the transition_state method in fsm/state_manager.py with optimistic concurrency control using cache-based locks, prevention of redundant same-state transitions, and correct mapping of entity fields for state model records. Organization context is now consistently set and logged. [1] [2] [3] [4]

FSM State and Transition Registration:

• Extended the register_state_transition decorator to support trigger metadata for transitions, allowing fine-grained control over when transitions are triggered (on create, update, or specific fields).
• Introduced a new AnnotationDraftStateChoices enum for draft annotation lifecycle states, supporting a more granular and independent workflow for annotation drafts.

Other Notable Improvements:

• Corrected annotation deletion logic to count items before deletion, ensuring accurate reporting of deleted annotations.

These changes collectively improve the reliability, observability, and flexibility of both context propagation and FSM-driven workflows across the application.

Most Important Changes:

Context Management & Async Job Context Propagation:

• Added CurrentContext for unified thread-local context management, replacing direct _thread_locals usage and providing methods for user, organization, and request handling.
• Async job submission now auto-captures and serializes context (user, organization, job data) for use in background jobs, ensuring proper context propagation. [1] [2]

FSM Infrastructure & Transition Handling:

• FSM state retrieval and transitions now respect a feature flag and include optimistic concurrency control, prevention of redundant transitions, and improved logging with organization context. [1] [2] [3] [4] [5] [6] [7]
• Enhanced register_state_transition decorator to support triggers on create, update, or specific fields, enabling more flexible FSM workflows.

FSM State Choices:

• Added AnnotationDraftStateChoices to support a full draft annotation lifecycle, independent from main annotation states.

Annotation Deletion:

• Fixed annotation deletion logic to count items before deletion for accurate reporting.